test(sync-service): reproduce FlushTracker stall when a tracked consumer dies without cleanup - #4713

…ected consumer lifecycle transitions (#4730)

## Problem

The FlushTracker pins `confirmed_flush_lsn` for the whole stack on the
minimum incomplete flush entry, and cleanup of entries is purely
event-driven: either new traffic routed to the same shape (publish-time
DOWN detection) or an explicit `ShapeLogCollector.remove_shape/2`
arriving via the ShapeCleaner chain. Several real-world paths break both
— a storage error that crashes the consumer *and* aborts its
`terminate/2` before `handle_writer_termination` runs, a removal chain
that dies between the `ShapeStatus` removal and the SLC removal
(unretryable: the retry short-circuits at `{:error, :data_removed}`),
untagged `{:shutdown, x}` exit reasons classified as benign, and a
consumer that stays alive but stops flushing. On a subsequently-quiet
table any of these pins the WAL ack forever, causing unbounded WAL
growth on Postgres. PR #4713 has a minimal reproduction of the stall.

## Solution

The SLC becomes the authority on consumer-death handling for flush
accounting. Invariant: an incomplete FlushTracker entry always has a
live monitor watching the pid responsible for completing it, and a
timestamp bounding how long it may sit without flush progress.

- **Writer monitors**: the SLC monitors every consumer with an
incomplete FlushTracker entry (created when a commit-fragment publish
tracks the shape, released when the entry completes or the shape is
removed). DOWNs are classified by exit reason: crashes unpin the flush
boundary immediately and trigger shape invalidation; deliberate-cleanup
DOWNs (`{:shutdown, :cleanup}`) unpin only; bare `:shutdown`/`:killed`
are assumed to be supervisor teardown and left alone.
- **Stall detection**: incomplete entries carry a `last_progress_at`
timestamp refreshed only by flush notifications. A periodic tick finds
shapes whose entries make no flush progress for
`ELECTRIC_FLUSH_STALL_GRACE_PERIOD` (default 60s) and invalidates them
via the normal ShapeCleaner path, re-arming on each fire so a lost
cleanup chain retries instead of pinning forever. Misclassified DOWNs
self-heal through the same mechanism.
- **Stall challenge-response**: a stalled entry whose writer is still
alive is challenged (`:verify_flush_progress`) before being acted on. A
consumer answers only while legitimately deferring flush notifications
(buffering ahead of PG snapshot info, subquery move-in awaiting splice);
the answer re-arms the entry's grace period so healthy deferral phases
are not misread as stalls. Only a suspect still stalled and silent at
the next check — or a shape whose writer is already dead and
demonitored, which skips the challenge — is invalidated. Consumers stay
silent in steady state: no periodic messaging, and a wedged consumer
can't answer, so the backstop stays armed.
- **Reason normalization**: the benign clause of
`handle_writer_termination` shrinks to exactly
`:normal`/`:killed`/`:shutdown`, so untagged `{:shutdown, x}` reasons
now trigger removal; tagged dependency-materializer shutdowns map to
`stop_and_clean`; suspend additionally requires all txn state flushed
and notified (guaranteeing a suspended consumer never leaves an
incomplete entry behind); `Consumer.stop` escalates a wedged consumer to
a kill after the stop call times out.
- ShapeCleaner re-issues the idempotent SLC removal even when the shape
is already gone from `ShapeStatus`, healing pins left by removal chains
that died partway through.

Telemetry: new `[:electric, :flush_tracker, :writer_down]` and
`[:electric, :flush_tracker, :stall_detected]` events.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Garry Hill <garry@electric-sql.com>